fix(temporal): retry Glitter startup metric restoration - #2006

• Add shutdown-aware equal-jitter retries for Glitter startup snapshot metric restoration.
• Retry only transient SeaweedFS/HTTP failures; escalate to Sentry after 10 consecutive failures.
• Add tests for retry delays, shutdown cancellation, and transient storage error classification.

graph TD
  A["Temporal worker"] --> B["retryUntilReady"] --> C["Restore snapshot metrics"] --> D[("SeaweedFS / S3")]
  B --> E["Structured logger"]
  B --> F{{"Sentry"}}

  subgraph Legend
    direction LR
    _svc["Service/module"] ~~~ _db[("Storage")] ~~~ _ext{{"External"}}
  end

Loading

The following are alternative approaches to this PR:

1. Mitigate via alert tuning (startup grace window)

• ➕ No code changes; reduces operational risk of retry loops
• ➕ Keeps worker startup simpler
• ➖ Weakens the alert signal for real snapshot issues during the grace period
• ➖ Does not address other startup consumers of SeaweedFS; pushes logic into monitoring

2. Infrastructure gating (init container / readiness on SeaweedFS)

• ➕ Prevents the race for all startup dependencies, not just metrics restoration
• ➕ Keeps application logic simpler
• ➖ Harder to express reliably across environments and failure modes
• ➖ May delay all worker readiness even if only corpus storage is impacted

3. Reuse existing supervisor pattern (linear backoff like event-bridge)

• ➕ Consistency with existing worker supervision approach
• ➕ Less new utility surface area
• ➖ Less robust than equal-jitter exponential backoff under contention
• ➖ Harder to unit-test deterministically without extracting a shared helper anyway

Recommendation: Proceed with the PR’s approach: a small, testable retry supervisor with explicit transient/permanent classification preserves alert semantics while eliminating known SeaweedFS startup races. The utility is generic enough to reuse (similar to the event-bridge supervisor), but remains scoped and shutdown-aware to avoid masking real failures.

Enhancement (1) +112 / -0

startup-retry.tsAdd shutdown-aware equal-jitter exponential backoff retry helper +112/-0

Add shutdown-aware equal-jitter exponential backoff retry helper

• Introduces 'retryUntilReady' plus 'equalJitterRetryDelayMs' and 'sleepUnlessClosed' to support indefinite startup retries with exponential ceiling and equal-jitter delays. Supports dependency injection for sleep/random and optional one-time escalation callback.

packages/temporal/src/shared/startup-retry.ts

Bug fix (2) +51 / -14

glitter-corpus-store.tsClassify transient corpus storage failures for safe retries +20/-0

Classify transient corpus storage failures for safe retries

• Introduces 'isTransientCorpusStorageError' to detect retryable failures via HTTP status codes (408/429/5xx) and common network error strings. Keeps existing not-found and precondition classification intact.

packages/temporal/src/activities/glitter-corpus-store.ts

worker.tsWrap Glitter startup metric restoration in transient-only retry loop +31/-14

Wrap Glitter startup metric restoration in transient-only retry loop

• Replaces the one-shot startup snapshot-metric restoration with 'retryUntilReady', retrying only when 'isTransientCorpusStorageError' matches. Logs every retry attempt, emits a single Sentry warning after 10 consecutive transient failures, and stops retrying once shutdown begins.

packages/temporal/src/worker.ts

Tests (2) +163 / -1

glitter-corpus-storage.test.tsAdd tests for transient SeaweedFS/S3 error classification +39/-1

Add tests for transient SeaweedFS/S3 error classification

• Imports and exercises the new transient error classifier across common connection error codes and retryable HTTP status codes. Verifies that permanent errors (auth/404 and malformed data) are not treated as retryable.

packages/temporal/src/activities/glitter-corpus-storage.test.ts

startup-retry.test.tsAdd deterministic tests for equal-jitter startup retry supervisor +124/-0

Add deterministic tests for equal-jitter startup retry supervisor

• Adds unit tests for equal-jitter delay calculation, delay capping, retry-until-success flow, shutdown cancellation, and single escalation behavior at the 10th consecutive transient failure.

packages/temporal/src/shared/startup-retry.test.ts

Documentation (1) +35 / -0

2026-08-07_glitter-corpus-startup-metric-retry.mdDocument plan for self-healing Glitter startup metric restoration +35/-0

Document plan for self-healing Glitter startup metric restoration

• Adds an in-progress plan describing indefinite equal-jitter retries for startup snapshot-metric restoration. Documents transient vs permanent failure handling, shutdown behavior, and verification steps.

packages/docs/plans/2026-08-07_glitter-corpus-startup-metric-retry.md

Description

isTransientCorpusStorageError only checks the top-level error’s $metadata.httpStatusCode and a regex
against ${error.name} ${error.message}, so a transient connection failure that is wrapped in an
Error.cause chain (where the outer error message doesn’t include the connection code) will be
treated as non-transient and will stop the startup retry loop.

Code

packages/temporal/src/activities/glitter-corpus-store.ts[R37-40]

+  return (
+    error instanceof Error &&
+    TRANSIENT_STORAGE_ERROR_PATTERN.test(`${error.name} ${error.message}`)
+  );

Evidence

The new classifier only matches transient connection codes in the outer Error’s name/message and
does not inspect .cause, while other code in this repo explicitly traverses .cause chains to
find the real underlying failure signal—meaning transient network failures can be hidden by wrappers
and incorrectly classified as non-transient.

packages/temporal/src/activities/glitter-corpus-store.ts[18-41]
packages/temporal/src/activities/data-dragon-util.ts[209-226]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`isTransientCorpusStorageError` ignores `Error.cause` and only inspects the outer error’s `name/message` for connection codes. If an SDK/network layer wraps the underlying connection error (common pattern in this repo), the transient signal may live in `cause` and retries won’t happen.

## Issue Context
There is already repo precedent for walking `.cause` chains to find the real failure message.

## Fix Focus Areas
- packages/temporal/src/activities/glitter-corpus-store.ts[23-41]
- packages/temporal/src/activities/data-dragon-util.ts[217-226]

## Suggested fix
- Build a helper to iterate `error` and its `.cause` chain (with cycle protection / depth limit), and for each link:
 - check `$metadata.httpStatusCode` (408/429/5xx)
 - check connection code patterns against combined text (e.g., `${name} ${message} ${stack ?? ""}`)
 - optionally check a common `code` field (e.g., `(err as any).code`) if present.
- Return true if any link matches transient criteria.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Description

worker.ts logs “Glitter corpus snapshot metric restoration completed” whenever retryUntilReady
returns "succeeded", but restoreGlitterCorpusSnapshotMetrics can return early without restoring
anything (not configured or latest pointer missing), making the log message inaccurate.

Code

packages/temporal/src/worker.ts[R117-119]

+    if (result === "succeeded") {
+      jsonLog("info", "Glitter corpus snapshot metric restoration completed");
+    }

Evidence

The worker logs completion when the retry wrapper reports success, but the restoration function
explicitly returns early in multiple cases where no restoration occurs, so the new log message
overstates what happened.

packages/temporal/src/worker.ts[87-120]
packages/temporal/src/activities/glitter-corpus-snapshot.ts[41-59]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The worker logs a completion message on any successful return from `restoreGlitterCorpusSnapshotMetrics`, but that function can legitimately do no work (e.g., metrics not configured or pointer missing). This makes logs less trustworthy during incident response.

## Issue Context
`restoreGlitterCorpusSnapshotMetrics` returns `void` and uses early returns for “skip” cases.

## Fix Focus Areas
- packages/temporal/src/worker.ts[91-119]
- packages/temporal/src/activities/glitter-corpus-snapshot.ts[41-59]

## Suggested fix
- Change `restoreGlitterCorpusSnapshotMetrics` to return a small status enum, e.g. `"restored" | "not_configured" | "pointer_missing"`.
- In `worker.ts`, log different messages (or include a field like `{ outcome }`) and only say “completed”/“restored” when metrics were actually updated.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools